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ABSTRACT 

Automated  systems  can  have  a  hard  time  completing  complex  tasks  in  a  timely  manner.  When 
controlling  a  robot  outside  of  autonomous  mode,  a  good  control  device  needs  to  give  the  user  full 
control  of  the  system  while  enabling  the  mission  to  be  completed  in  a  quick,  accurate  and  efficient 
manner.  This  paper  outlines  the  potential  features  of  a  puppet  style  control  device  and  the  lessons 
learned  while  implementing  such  a  device. 


INTRODUCTION 

As  ground  robotics  moves  towards  autonomous  and  semi- 
autonomous  operations,  the  need  to  have  full  control  over 
manipulators  is  still  required  for  complex  situations  or  where 
a  user  feels  the  need  to  take  control  of  the  system.  Repetitive 
tasks  tend  to  be  the  tasks  where  humans  begin  to  feel 
fatigued  and  make  mistakes.  Automation  is  often  great  in 
these  situations.  Complex  tasks  keep  a  human  alert  and 
thinking  as  long  as  the  task  can  be  performed  in  a  short 
amount  of  time.  Automated  systems  tend  to  have  a  harder 
time  completing  these  complex  tasks  accurately  and  in  a 
timely  manner.  Our  research  has  shown  that  when 
controlling  a  robot  outside  of  autonomous  mode,  a  good 
control  device  needs  to  give  the  user  full  control  of  the 
system  while  enabling  the  mission  to  be  completed  in  a 
quick,  accurate  and  efficient  manner  [1].  Additional  research 
shows  that  limitations  in  a  control  device  can  often  reduce 
the  usefulness  of  the  robot  [2]. 

In  recent  years,  manipulators  have  become  more  capable 
and  more  dexterous  through  additional  degrees  of  freedom. 
As  a  result,  new  control  techniques  and  new  control  devices 
have  been  required.  Techniques  such  as  “flying  the  end 
effector”  require  little  cognitive  load  when  working  in 
environments  with  a  minimal  amount  of  obstacles,  but  users 
do  not  have  control  of  all  of  the  joints  and  links  with  this 
control  method.  This  can  potentially  result  in  unwanted 
configurations  of  the  robot  or  collisions  with  obstacles. 
Current  methods  for  having  full  control  over  all  of  the  joints 


often  requires  moving  one  or  two  joints  at  a  time  through  the 
use  of  knobs  or  buttons,  resulting  in  an  overall  system  that  is 
hard  to  control,  very  slow  and  is  tedious  for  the  operator. 

RE2  in  conjunction  with  TARDEC,  through  an  SBIR 
project  called  Modular  Intuitive  Manipulation  with  Intuitive 
Control  (MIMIC),  set  out  to  research  a  wide  range  of 
existing  control  devices  with  the  goal  of  determining  which 
devices  were  the  most  intuitive  to  use  and  resulted  in  the 
fastest  and  highest  success  rate  for  common  tasks.  This 
research  also  looked  into  combining  devices  to  utilize  the 
positive  aspects  of  the  devices  while  canceling  out  the 
negative  aspects.  Phase  I  of  the  project  looked  at  existing 
products  and  techniques  to  determine  their  strengths  and 
weaknesses  in  terms  of  ease  of  control  and  their  extensibility 
to  additional  degrees  of  freedom,  among  other  factors.  This 
research  identified  several  approaches  to  manipulation 
control  of  high  degree  of  freedom  arms.  Under  the  resulting 
Phase  II  effort,  the  MIMIC  program  performed  in-depth 
testing  and  analysis  of  the  most  promising  devices  with  a 
wide  range  of  users.  These  prototypes  and  concepts  led  to 
further  research  and  development  of  the  most  promising 
solution  -  a  puppet  controller  -  through  a  contract  with  the 
U.S.  Navy.  This  new  control  device,  called  Imitative 
Controller  (IC),  allows  the  user  to  move  a  scaled  model, 
sometimes  referred  to  as  a  puppet,  of  the  robot’s 
manipulators.  This  controller  has  been  shown  to  effectively 
control  a  highly  dexterous  two-arm  system,  consisting  of 
two  7  degrees-of-freedom  arms  and  a  2  degrees-of-freedom 
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torso  (16  total  degrees-of- freedom),  developed  though  a 
contract  with  the  Office  of  Naval  Research  entitled 
Dexterous  Manipulation  System  (DMS). 

PUPPETS 


Figure  1:  Imitative  Controller  (top)  in  a  configuration 
set  to  be  a  scaled  down  match  of  the  joints  and  links  of 
the  HDMS  robot  (bottom) 

A  puppet  control  device  is  often  comprised  of  joints  and 
links  that  are  to  scale  of  the  robot  that  the  device  is 
controlling.  The  joints  have  encoders  in  them  to  determine 
the  position  and  orientation  of  the  device,  which  is  then 
translated  to  movement  and  position  commands  for  the 
robot.  In  the  case  of  IC  (Figure  1,  top),  the  user  holds  onto  a 
handle  and  moves  their  hand  around,  which  moves  all  of  the 
joints  and  links  between  the  handle  and  the  base  of  the 
device.  Given  that  the  puppet  device  is  a  model  of  the  robot, 
the  robot  will  attempt  to  match  the  position  and  orientation 
in  a  master/slave  relationship.  This  point  of  interaction 
between  the  human  and  the  control  device  is  similar  to  that 
which  is  sometimes  used  for  flying  the  end  effector 
(Monakhova,  2013),  as  seen  in  Figure  2. 


Figure  2:  Control  devices  that  fly  the  end  effector  can 
result  in  the  robot  moving  to  unintended  orientations 

In  this  respect,  the  user  is  using  the  device  in  a  similar 
manner,  but  with  a  puppet  device  they  get  a  physical 
confirmation  that  the  overall  state  of  the  robot  will  consist  of 
a  set  of  specific  positions  and  orientations.  Flying  the  end 
effector  relies  on  mathematical  formulas  which  can  produce 
multiple  solutions  to  the  same  desired  position  and 
orientation  resulting  in  an  unknown  final  state  of  the  robot 
even  when  the  position  and  orientation  of  the  end  effector  is 
known. 

Handles 

Our  most  recent  research  has  centered  around  handle 
types,  handle  locations,  and  the  overall  size,  or  scaling 
factor,  of  puppet  systems  in  order  to  optimize  the  user’s 
experience  and  ensure  the  most  accurate  and  intuitive 
control  of  the  robot. 

Handle  types  refer  to  the  style  of  grip  that  is  intended  as 
the  connection  point  between  the  user  and  the  control 
device.  We  have  found  that  the  best  handles  are  those  that 
are  designed  to  fit  in  the  user’s  hand  in  such  a  manner  that  it 
is  at  an  angle  to  the  piece  of  the  control  device  that  is  being 
controlled,  such  as  the  grip  on  the  left  hand  side  of  Figure  3. 

The  curvature  of  the  grip  between  the  thumb  and 
forefinger  also  appears  to  aid  in  better  control  and  more 
comfort  for  the  user.  This  style  handle  also  allows  for 
several  easy  to  reach  buttons  to  be  added.  Straight  “bicycle 
handle  bar”  style  handles  cause  extra  strain  on  the  user’s 
wrists  and  is  especially  uncomfortable  when  buttons  are 
added  since  the  user’s  fingers  need  to  stretch  and  assert  force 
in  an  unnatural  angle. 

We  have  found  the  location  of  the  handle  works  best  when 
it  is  located  close  to  the  end  point  of  the  control  device.  This 
allows  the  user  to  intuitively  control  the  robotic  arms  since 
the  user’s  hands  are  in  a  similar  location  as  to  where  the 
robot’s  end  effectors  are  located.  This  especially  becomes 
important  when  controlling  a  two  armed  system  and 
coordinated  movement  between  the  two  arms  is  required, 
such  as  when  handing  an  object  from  one  end  effector  to 
another,  unscrewing  an  object  while  also  holding  the  base  of 
that  object,  or  performing  complex  multi-end-effector 
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manipulation  such  as  tying  a  knot.  When  the  location  of  the 
handle  was  moved  further  up  the  control  device  (away  from 
the  end  point),  the  user  took  longer  to  line  up  the 
manipulator  to  the  object  that  was  being  manipulated  and 
had  a  higher  “miss”  rate  where  the  user  misjudged  the 
location  of  the  end  effector  compared  to  the  object  that  they 
intended  to  manipulate.  Handle  locations  are  also  important 
since  humans  can  only  bend  and  twist  their  own  bodies  in 
certain  directions  and  to  certain  extents,  often  proving  to  be 
the  limiting  factor  for  the  range  of  motion  in  the  system. 
Proper  handle  locations  ensure  full  control  of  the  device 
while  providing  an  intuitive  method  for  doing  any  given 
task. 


Figure  3:  A  grip  with  a  handle  that  is  at  an  angle  offers 
better  ergonomics  to  the  user  when  comared  to  straight 
handles  (right) 


Links  and  Joints 

A  proper  scaling  factor  for  a  puppet  system  is  important  to 
ensure  users  can  comfortably  control  the  manipulators  using 
the  natural  motion  of  their  arms  and  hands.  Link  lengths  that 
are  too  long  result  in  the  user  stretching  to  reach  certain 
positions  and  the  user  will  become  fatigued  faster.  If  the  link 
lengths  are  too  short,  the  user’s  workspace  gets  smaller  and 
can  increase  the  difficulty  of  controlling  the  robot.  A  good 
scaling  factor  needs  to  be  determined  for  each  individual 
robot  and  controller  pair.  This  becomes  a  hard  problem  to 
solve  as  users  may  have  different  length  arms,  making  it 
hard  to  find  a  good  fit  for  all  potential  users.  It  is  also 
important  to  keep  the  same  scale  factor  throughout  the 
puppet  device.  Failing  to  do  so  will  violate  the  puppet  nature 
of  the  controller  and  the  end  effector  position  of  the 
manipulator  will  not  match  the  position  of  the  controller. 

The  design  of  the  robot  is  also  a  factor  when  choosing  a 
scale  factor.  If  several  small  joints  are  clustered  in  one  area 
of  the  robot  and  then  the  rest  of  the  robot  is  large  and  sparse 
between  joints,  a  scaled  down  puppet  device  may  be  hard  to 
achieve  due  to  the  small  size  needed  to  keep  the  proper 
scaling  near  the  cluster  of  joints. 

To  further  ensure  the  puppet  system  matches  the 
capabilities  of  the  robot,  the  joint  limits  on  the  puppet  device 
should  match  those  of  the  robot.  In  addition  to  ensuring  the 
range  of  motion  is  the  same  between  the  puppet  and  the 
robot,  joint  limits  act  as  a  physical  feedback  mechanism  to 
the  user  to  let  them  know  they  have  reached  the  joint  limits. 


Base 

The  base  of  the  puppet  can  take  many  different  forms.  The 
one  that  is  best  is  dependent  on  the  application  as  well  as 
how  and  where  the  user  intends  to  use  the  puppet.  Fast 
movements  call  for  a  stable  base,  such  as  on  a  tripod  or 
otherwise  attached  to  a  heavy,  stable  object  (such  as  a  table 
or  the  chair  the  user  is  sitting  in).  Slower  movements  can  be 
less  demanding  on  the  base  of  the  control  device,  however  it 
still  requires  a  base  that  is  heavy  enough  or  to  be  otherwise 
attached  to  something  sturdy  to  stay  in  place  when  the  joints 
are  being  moved. 

FEEDBACK 

Feedback  to  the  user  enables  the  user  to  better  understand 
the  interaction  between  the  robot  and  the  environment 
around  the  robot.  Feedback  can  inform  a  user  how  tightly  the 
robot  is  grasping  an  object,  if  the  robot  has  bumped  into 
something,  or  to  simply  give  the  user  a  status  update. 
Feedback  can  be  expressed  in  several  different  methods, 
such  as  light,  sound  or  force. 

Light 

Light  feedback  can  also  be  further  broken  down  into 
several  different  modalities.  A  simple  light,  such  as  a  single 
LED,  can  change  colors  to  indicate  feedback.  It  can  blink 
and  change  the  speed  at  which  it  blinks.  If  the  light  source  is 
on  a  movable  platform  or  if  there  are  several  lights  spread 
out  across  a  specified  area,  the  location  of  the  lights  can  also 
offer  feedback. 

For  example,  imagine  a  row  of  4  LEDs  that  are  located  on 
the  handle  of  a  control  device.  These  4  LEDs  could  indicate 
the  location  of  the  robot’s  gripper  in  terms  of  how  open  or 
closed  the  gripper  is.  A  separate  LED  could  change  colors 
from  green  to  yellow  to  red  to  indicate  how  hard  the  gripper 
is  grasping  an  object.  This  same  LED  can  start  to  blink  if  the 
robot  detects  that  the  object  is  starting  to  slip  out  of  its  grasp. 
This  simple  visual  feedback  will  offer  the  user  valuable 
information  that  can  increase  the  user’s  ability  to 
successfully  complete  the  task  at  hand. 

Sound 

Similar  to  light  based  feedback,  sound  can  offer  simple 
feedback  to  the  user.  A  user  is  often  using  their  sight  to 
control  a  robot.  Asking  the  user  to  divert  their  gaze  to  an 
LED,  even  for  a  fraction  of  a  second,  could  result  in  a  failed 
mission.  Pulsing  sounds,  such  as  a  “Geiger  counter”  noise, 
can  allow  the  user  to  keep  their  visual  attention  on  the  task 
while  adding  to  the  user’s  knowledge  by  this  repetition  of 
noise. 

An  example  of  the  use  of  sound  can  include  using  a 
repeating  sound  that  repeats  faster  as  the  robot’s  gripper  gets 
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closer  to  an  object  and  slower  as  the  gripper  moves  away 
from  the  object. 

Haptic 

Our  earlier  research  found  haptic  feedback  to  be  the  most 
useful  of  the  feedback  types  when  manipulating  objects. 
Haptic  feedback  can  come  in  a  variety  of  types  including 
vibration  and  through  resistance  by  pushing  or  pulling 
against  the  user. 

Vibration  feedback  can  be  accomplished  through  small 
vibrating  motors  such  as  those  found  in  modem  cell  phones. 
The  placement  and  intensity  of  these  motors  is  important  to 
ensure  the  user  feels  the  device  vibrating  [3],  but  also  to 
ensure  the  device  does  not  vibrate  so  much  as  to  distract  the 
user  from  the  task. 

Force  feedback  can  be  accomplished  by  giving  the  puppet 
device  motors  in  the  joints  or  in  the  portion  of  the  device 
that  controls  the  robot’s  grippers.  The  motors  can  attempt  to 
hold  a  specific  position,  or  push  back  in  a  certain  direction, 
once  a  signal  from  the  robot  is  received  indicating  the  robot 
is  grasping  an  object,  one  of  the  joints  cannot  move  in  a 
specific  direction,  or  for  any  other  number  of  reasons.  This 
resistance  prevents  further  movement  in  the  wrong  direction 
and  informs  the  user  that  further  movement  in  that  direction 
is  not  possible. 

PROS  AND  CONS 

A  puppet  control  device,  consisting  of  a  scaled  model  of  a 
robot,  allows  the  user  to  have  very  specific  control  over  all 
the  joints  of  that  robot  while  providing  an  interface  that  is 
intuitive  and  easy  to  use.  Through  this  method  of 
manipulation,  a  control  device  can  contain  more  degrees  of 
freedom  than  the  user  has  on  their  own  arm.  Although  this 
sounds  counter-intuitive,  the  additional  degrees  of  freedom 
are  controlled  through  the  help  of  biased  movements  of  the 
joints.  For  example,  a  bending  joint  may  be  biased  to  bend  in 
one  direction  more  so  than  in  the  other  direction.  This  bias 
can  be  easily  overcome  by  the  user  if  they  wish  a  joint  to 
bend  in  a  particular  manner,  but  otherwise  the  decision  on 
which  way  to  bend  is  pre-determined.  So  even  though  the 
operator  can  only  actively  control  the  six  degrees  of  freedom 
of  the  handle  of  the  controller,  the  biasing  helps  control  the 
additional  degrees  of  freedom  This  is,  in  part,  what  solves 
the  drawbacks  with  the  “flying  the  end  effector”  design  and 
can  contribute  towards  making  a  puppet  control  device  more 
intuitive  to  use. 

There  are  drawbacks  to  using  a  puppet  device  too.  Often 
an  entirely  new  control  device  is  needed  that  is  specific  to 
the  robot  under  control.  Specializing  controllers  in  this 
manner  often  means  higher  cost.  By  having  a  modular 
architecture,  similar  to  the  one  developed  for  IC,  links  can 
be  cut  to  size  and  joints  are  non-specific.  This  lowers  the 
cost  of  creating  a  novel  configuration  for  a  given  robot  and 


makes  it  more  available  to  a  wider  range  of  customers  and 
applications. 

TESTING 

Through  previous  phases  of  this  project  and  through  other 
related  projects,  RE2  has  evaluated  various  control  devices. 
For  comparative  means,  some  tasks  are  common  across 
these  tests  to  help  determine  the  feasibility  of  a  robotic 
system.  For  the  purposes  of  this  section,  we  will  consider  a 
robotic  system  to  be  a  combination  of  the  robot  and  the 
control  device  that  is  being  used  to  control  the  robot. 
Previous  tests  paired  existing  control  devices  with  existing 
robot  systems,  but  neither  was  designed  with  the  other 
device  in  mind  -  software  was  written  to  bridge  that  gap. 
The  latest  results  of  our  testing  focused  on  Imitative 
Controller  (IC)  paired  with  RE2’s  Highly  Dexterous 
Manipulation  System  (HDMS).  The  IC  device  is  not 
specifically  designed  to  work  with  HDMS,  but  is 
configurable  to  match  an  existing  (or  future)  robotic 
manipulator  by  adjusting  the  link  lengths  of  the  IC  device. 
This  adjustment  was  made  before  this  round  of  testing. 

One  test  that  was  performed  in  this  round  of  evaluation 
included  picking  up  a  tube  and  placing  it  in  a  box  of  roughly 
the  same  size  as  the  tube.  The  tube  is  a  semi  compliant 
plastic  cylindrical  container  measuring  roughly  8  inches  tall 
with  a  diameter  of  3.25  inches,  as  shown  in  Figure  4.  The 
box  is  made  out  of  cardboard  and  is  open  at  the  top.  The 
inside  of  the  box  is  divided  into  a  2x3  grid  of  4  inch  by  4 
inch  compartments  and  is  roughly  13  inches  tall.  A  small 
table  was  used  to  place  these  objects  within  the  robot’s 
workspace. 

Users  were  told  which  of  the  box’s  six  compartments  they 
had  to  place  the  tube  into  using  the  IC  control  device  and  the 
HDMS  robot.  The  box  was  not  attached  to  the  table  and 
could  shift  its  location  if  bumped  by  the  arm.  If  the  user 
knocked  the  tube  off  the  table  or  otherwise  dropped  it  while 
attempting  to  place  it  into  the  box,  both  the  robot  and  the 
timer  were  paused  while  the  proctor  of  the  test  retrieved  the 
tube.  The  tube  was  placed  back  onto  the  table  in  its  original 
starting  location,  then  the  robot  and  timer  resumed  when  the 
user  continued  the  task.  Figure  4  shows  the  robot  dropping 
the  tube  into  the  box. 
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Figure  4:  A  robot  dropping  a  tube  into  a  box  as  part 
of  a  timed  test 


The  IC  device,  paired  with  HDMS,  outperformed  other 
previously  tested  non-puppet  control  devices  in  this  task  by 
over  50  seconds. 


Controller 

Canister  Test  Time  (seconds) 

Non-puppet  device 

75 

Puppet  device 

56 

IC  device 

24 

This  allows  the  device  to  match  several  different  models  of 
manipulators  with  minimal  effort. 

The  simplicity  of  puppet  devices  offers  the  ability  to 
control  complex  manipulators  and  has  sparked  the  interest  of 
various  programs  and  fields.  In  a  project  for  the  Office  of 
Naval  Research,  a  two  armed  underwater  robot  with  7  DOF 
for  each  arm  is  being  developed.  The  ability  to  control  such 
a  system  would  be  nearly  impossible  with  techniques  such  as 
individual  joint  control.  EOD  and  bomb  squads  have  shown 
interest  in  using  a  puppet  control  device  to  control  existing 
robots,  such  as  the  PackBot  and  the  Remotec  F6A.  Home 
healthcare  is  also  an  industry  where  this  type  of  control 
shows  potential.  A  simple  control  mechanism,  such  as  IC, 
can  enable  a  person  in  a  wheelchair  to  control  a  complex 
robotic  arm  to  help  lift  items  off  the  floor,  reach  up  high  into 
cabinets,  open  doors  or  carry  heavy  items. 
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Other  tests  that  were  performed  include  removing  a 
blasting  cap  from  a  block  of  clay,  reaching  behind  and 
around  a  box  to  grasp  an  object,  and  pressing  a  series  of  6 
elevator  style  buttons  in  a  specified  order.  In  these  test,  IC 
was  consistently  two  to  three  times  faster  than  previously 
tested  systems. 

CONCLUSION 

When  done  properly,  puppet  systems  have  the  benefits  of 
the  “flying  the  end  effector”  technique,  but  maintain  full 
control  over  all  the  joints  and  links  of  the  robot.  This 
combination  allows  the  user  to  intuitively  and  easily  adjust 
all  of  the  joints  of  a  manipulator,  whether  it  is  3,  7,  or  any 
other  number  of  joints  on  a  single  or  dual  arm  system. 
Puppet  systems,  though,  have  the  down  side  of  only  being 
useful  for  the  robot  that  the  puppet  is  designed  to  control.  To 
counteract  this  shortcoming,  these  control  devices  can  be 
designed  to  be  easily  reconfigurable  for  changes  in  the 
number  of  joints,  link  lengths  and  offsets  between  joints. 


Imitative  Robotic  Control:  The  Puppet  Master,  Rusbarsky,  et  al. 


Page  5  of  5 
UNCLASSIFIED 


